Skip to content

🐛Storage: Listing path query grows with number of projects unbounded #8542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Oct 21, 2025
Merged

🐛Storage: Listing path query grows with number of projects unbounded #8542

merged 3 commits into from
Oct 21, 2025
+27 −11

Conversation

@sanderegg
Copy link
Member

@sanderegg sanderegg commented Oct 21, 2025 •
edited
Loading

What do these changes do?

fixes
image

that happens when the number of accessible projects grows beyond some number.

details

the cursor pagination parameter was integrating the list of projects to filter with and this growths with the number of projects.
luckily the e2e user projects never get deleted and that is why this started to happen.

the fix

Related issue/s

How to test

Dev-ops

@sanderegg sanderegg added this to the Imparable milestone Oct 21, 2025
@sanderegg sanderegg self-assigned this Oct 21, 2025
@sanderegg sanderegg added bug buggy, it does not work as expected a:storage issue related to storage service t:maintenance Some planned maintenance work labels Oct 21, 2025
@codecov
Copy link

codecov bot commented Oct 21, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.25%. Comparing base (0b2ea9f) to head (a1ee341).
⚠️ Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (0b2ea9f) and HEAD (a1ee341). Click for more details.

HEAD has 26 uploads less than BASE
Flag BASE (0b2ea9f) HEAD (a1ee341)
unittests 27 1
Additional details and impacted files 
@@             Coverage Diff             @@
##           master    #8542       +/-   ##
===========================================
- Coverage   80.97%   67.25%   -13.73%     
===========================================
  Files        1833      838      -995     
  Lines       71465    38121    -33344     
  Branches     1344      175     -1169     
===========================================
- Hits        57872    25637    -32235     
+ Misses      13205    12427      -778     
+ Partials      388       57      -331
Flag Coverage Δ
integrationtests 63.91% <ø> (+2.87%) ⬆️
unittests 86.84% <100.00%> (+6.00%) ⬆️
Components Coverage Δ
pkg_aws_library ∅ <ø> (∅)
pkg_celery_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 76.81% <ø> (+11.62%) ⬆️
agent ∅ <ø> (∅)
api_server ∅ <ø> (∅)
autoscaling ∅ <ø> (∅)
catalog ∅ <ø> (∅)
clusters_keeper ∅ <ø> (∅)
dask_sidecar ∅ <ø> (∅)
datcore_adapter ∅ <ø> (∅)
director ∅ <ø> (∅)
director_v2 77.96% <ø> (-7.38%) ⬇️
dynamic_scheduler ∅ <ø> (∅)
dynamic_sidecar 81.89% <ø> (ø)
efs_guardian ∅ <ø> (∅)
invitations ∅ <ø> (∅)
payments ∅ <ø> (∅)
resource_usage_tracker ∅ <ø> (∅)
storage 86.84% <100.00%> (∅)
webclient ∅ <ø> (∅)
webserver 58.90% <ø> (-13.21%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b2ea9f...a1ee341. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sanderegg sanderegg requested a review from Copilot October 21, 2025 14:28
Copilot AI reviewed Oct 21, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR addresses a bug where listing path queries grew unboundedly with the number of projects. The fix removes project_ids from the pagination cursor to prevent it from growing without bounds, since cursors are passed through REST APIs. Instead, the project_ids filter is passed directly as a function parameter with validation to ensure it doesn't exceed reasonable limits.

Key Changes:

  • Removed project_ids from the pagination cursor structure to prevent unbounded growth
  • Added input validation with a maximum length constraint (10,000) on filter_by_project_ids
  • Added debug logging for S3 listing cursor operations

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
services/storage/src/simcore_service_storage/utils/simcore_s3_dsm_utils.py Added debug logging for cursor operations when listing child paths in S3
services/storage/src/simcore_service_storage/modules/db/file_meta_data.py Removed project_ids from cursor parameters, added validation with max length constraint, and updated references to use the parameter directly

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@sonarqubecloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@sanderegg sanderegg marked this pull request as ready for review October 21, 2025 14:32
pcrespov
pcrespov approved these changes Oct 21, 2025
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx!

odeimaiz
odeimaiz approved these changes Oct 21, 2025
View reviewed changes
Copy link
Member

@odeimaiz odeimaiz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merci!

matusdrobuliak66
matusdrobuliak66 approved these changes Oct 21, 2025
View reviewed changes
Copy link
Collaborator

@matusdrobuliak66 matusdrobuliak66 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍 yes we can sometimes check the join together

@mergify
Copy link
Contributor

mergify bot commented Oct 21, 2025 •
edited
Loading

🧪 CI Insights

Here's what we observed from your CI run for a1ee341.

✅ Passed Jobs With Interesting Signals

Pipeline Job Signal Health on master Retries 🔍 CI Insights 📄 Logs
CI system-tests Base branch is broken, but the job passed. Looks like this might be a real fix 💪 Broken 0 View View

@sanderegg sanderegg merged commit 36d55e9 into ITISFoundation:master Oct 21, 2025
93 of 95 checks passed
@sanderegg sanderegg deleted the bugfix/storage-get-path-cursor-growing branch October 21, 2025 14:57
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Oct 28, 2025
Open
22 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@GitHK GitHK GitHK left review comments

Copilot code review Copilot Copilot left review comments

@pcrespov pcrespov pcrespov approved these changes

@odeimaiz odeimaiz odeimaiz approved these changes

@matusdrobuliak66 matusdrobuliak66 matusdrobuliak66 approved these changes

@giancarloromeo giancarloromeo Awaiting requested review from giancarloromeo

@bisgaard-itis bisgaard-itis Awaiting requested review from bisgaard-itis

Assignees

@sanderegg sanderegg

Labels

a:storage issue related to storage service bug buggy, it does not work as expected t:maintenance Some planned maintenance work

Projects

None yet

Milestone

Imparable

Development

Successfully merging this pull request may close these issues.

5 participants

@sanderegg @GitHK @pcrespov @odeimaiz @matusdrobuliak66